FRVT 1:1 Verification

Latest Report (June 28, 2021) | Participation Agreement | API | Validation | Encryption

Status

[Update: March 23, 2021] - The 1:1 matching time limit in Table 1.3 of the API has been updated from 5 milliseconds to 0.1 milliseconds (or 100 microseconds). For developers that cannot meet the new required time limit for matching two templates please contact

All FRVT tracks are open. Algorithms submitted to the FRVT 1:1 track will be run on all the datasets documented in the FRVT 1:1 report. We will additionally run the algorithms on photos in which a face may be occluded to simulate the effect of wearing a protective face mask - see FRVT Face Mask Effects webpage for more information.


FRVT Participation Statistics

[last updated: July 14 , 2021]

2017 2018 2019 2020 2021 Total
Number of algorithms 58 101 167 140 97 563
Number of unique developers 31 50 109 105 92 218

FRVT 1:1 Performance Summary

[last updated: July 14 , 2021]

Accuracy

Verification Performance

The table shows the top performing 1:1 algorithms measured on false non-match rate (FNMR) across several different datasets. FNMR is the proportion of mated comparisons below a threshold set to achieve the false match rate (FMR) specified. FMR is the proportion of impostor comparisons at or above that threshold. The algorithms are ordered initially in terms of rank on the VISABORDER dataset, calculated using FNMR @ FMR = 10-6. The rows for a given column may be sorted by clicking on its header. Hovering over the algorithm will show you the corresponding company name. In addition, searching can be done using regular expressions and company names are searchable.

Resources

Resources Performance

Algorithms submitted to FRVT implement NIST’s application programming interface (API). We measure the duration of all function calls using the C++ std::chrono::high resolution clock on an unloaded server-class machine. The table below includes durations of the template generation, finalization, search calls. In addition the size of the algorithm is reported in two parts: the recognition models, and the libraries.

## Warning in read.table(file = file, header = header, sep = sep, quote = quote, :
## cols = 14 != length(data) = 15

Additional algorithms not listed in the table can be found in Table 2 of our latest FRVT report. It is a draft made available for comments. It will be updated on a monthly basis as algorithms and computations complete, as datasets are added, and as new analyses are developed.

Prior Editions of Report

All prior Ongoing FRVT 1:1 reports can be accessed from here.

Overview

NIST started a new evaluation of face recognition technologies in February 2017. Unlike previous evaluations, the activity is conducted on an ongoing basis in that the evaluation remains open indefinitely such that developers may submit their algorithms to NIST whenever they are ready, but no more frequently than four calendar months. The algorithms will be evaluated rapidly on a first-come-first-served basis, following our MINEX III evaluation of fingerprint recognition implementations. Performance results will be posted to the NIST website as soon as they are ready. This approach more closely aligns evaluation with development schedules; this improves over the two to four year interval between past FRVT tests.

Goals

The FRVT is aimed at measurement of the performance of automated face recognition technologies applied to a wide range of civil, law enforcement and homeland security applications including verification of visa images, de-duplication of passports, recognition across photojournalism images, and identification of child exploitation victims. In all cases the input image will contain one face only. Our performance reports will include measurements of accuracy, speed, storage and memory consumption, and resilience. NIST will report the dependence of performance on the properties of the images and the subjects. In its initial form, FRVT has one assessment track, for face verification.

How to Participate

To participate in this evaluation, developers need to submit a participation agreement to NIST, wrap their software behind the published C++ API, run their libraries through the provided validation package (which creates a submission package), encrypt the package, and provide a download link for the encrypted submission package. More details are provided below.

Participation Agreement

FRVT is conducted by NIST, an agency of the United States Government. Participation is free of charge. FRVT is open to a global audience of face recognition developers. All organizations who seek to participate in FRVT must sign and submit all pages of this Participation Agreement by mail AND send a PDF scanned copy to frvt@nist.gov. [last update: 2020-06-15]

Note: In addition to the terms of the Participation Agreement 1) all correspondence from the provider to NIST must come from an organizational email address of the participating entity – no generic emails from gmail.com, mail.ru, qq.com, etc. will be accepted; 2) the point of contact listed in the participation agreement must specify an organizational email address; 3) the organization must have a non-trivial web presence. [last update: 2018-01-09]

API Document

A new API document has been published. All FRVT APIs reference the supporting FRVT General Evaluation Specifications, which includes hardware and operating system environment, software requirements, reporting, and common data structures that support the APIs. [last update: 2021-03-22]

Validation

An updated validation package has been published. All participants must run their software through the updated validation package prior to submission. The purpose of validation is to ensure consistent algorithm output between your execution and NIST’s execution. Note: The provider identifier in the core implementation library must be an appropriate, representative, non-infringing name of the main provider of the software. If the provider identifier is not representative of the submitting organization, NIST reserves the right to reject the submission for testing. [last update: 2020-09-10]

Encryption

All submissions must be properly encrypted and signed before transmission to NIST. This must be done according to these instructions using the FRVT Ongoing public key linked from this page. Participants must email their public key to NIST. The participant’s public key must correspond to the participant’s public-key fingerprint provided on the signed Participation Agreement. [last update: 2017-09-11]

Submission

Encrypted files below 20MB can be emailed to NIST at frvt@nist.gov. Encrypted files above 20MB can be provided as a download link from a generic http webserver (e.g., Google Drive). We cannot accept Dropbox links. NIST will not register, or establish any kind of membership, on the provided website. Participants can submit their algorithm(s) as soon as the signed participation agreement is sent to NIST. There is no need to wait for NIST confirmation of the received agreement. Participants must subscribe to the FRVT mailing list to receive emails when new reports are published or announcements are made. [last update: 2020-03-27]

Contact Information

Inquiries and comments may be submitted to frvt@nist.gov.

Subscribe

Subscribe to the FRVT mailing list to receive emails when announcements or updates are made.

Related NIST Projects

FRVT Ongoing
FRVT 1:N Identification
FRVT Demographic Effects
FRVT MORPH
FRVT Quality Assessment
FRVT Face Mask Effects
FRVT Paperless Travel